from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(against_lib="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	2.089471	0.179977	NaN	...	brute	-1	1	0.663	0.310865	0.007162	1.000	6.721481	6.723264	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.025118	0.002184	NaN	...	brute	-1	1	1.000	17.124995	0.183943	0.757	0.001467	0.001467	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	2.932399	0.052733	NaN	...	brute	-1	5	0.757	16.969067	0.010442	0.882	0.172808	0.172809	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	2.060044	0.005000	NaN	...	brute	1	100	0.882	0.317820	0.004952	1.000	6.481800	6.482587	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.021647	0.000338	NaN	...	brute	1	100	1.000	17.019915	0.055487	0.757	0.001272	0.001272	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	2.983173	0.040908	NaN	...	brute	-1	100	0.882	17.369627	0.046000	0.663	0.171747	0.171747	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	2.052457	0.007466	NaN	...	brute	1	5	0.757	0.236219	0.003240	1.000	8.688776	8.689593	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.020573	0.000466	NaN	...	brute	1	5	1.000	3.708129	0.008333	0.922	0.005548	0.005548	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.194637	0.011474	NaN	...	brute	1	1	0.663	3.748850	0.015796	0.929	0.318668	0.318671	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.865892	0.016249	NaN	...	brute	-1	1	0.896	0.239038	0.004342	1.000	7.805847	7.807134	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.006118	0.004813	NaN	...	brute	-1	1	1.000	3.718485	0.014296	0.922	0.001645	0.001645	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.716382	0.017446	NaN	...	brute	-1	5	0.922	3.752791	0.007130	0.896	0.723830	0.723831	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.001	6.860	0.0	-1	1	17.330	0.040	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.828	0.0	-1	5	0.314	0.005	1.000	0.037	0.037	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.827	0.0	1	100	17.341	0.032	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.012	0.000	6.892	0.0	-1	100	0.315	0.006	1.000	0.037	0.037	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	6.970	0.0	1	5	3.765	0.008	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.018	0.0	1	1	0.237	0.004	1.000	0.048	0.048	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	0.374	0.0	-1	1	3.762	0.026	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	0.375	0.0	-1	5	0.238	0.004	1.000	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.089	0.180	0.000	0.002	-1	1	0.311	0.007	1.000	6.721	6.723	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.002	0.000	0.025	-1	1	17.125	0.184	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.932	0.053	0.000	0.003	-1	5	16.969	0.010	0.882	0.173	0.173	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.002	0.000	0.025	-1	5	0.317	0.005	1.000	0.078	0.078	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.060	0.005	0.000	0.002	1	100	0.318	0.005	1.000	6.482	6.483	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.022	0.000	0.000	0.022	1	100	17.020	0.055	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.983	0.041	0.000	0.003	-1	100	17.370	0.046	0.663	0.172	0.172	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.004	0.000	0.026	-1	100	0.319	0.005	1.000	0.082	0.082	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.052	0.007	0.000	0.002	1	5	0.236	0.003	1.000	8.689	8.690	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.000	0.000	0.021	1	5	3.708	0.008	0.922	0.006	0.006	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.195	0.011	0.001	0.001	1	1	3.749	0.016	0.929	0.319	0.319	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.002	0.000	0.020	1	1	0.239	0.005	1.000	0.085	0.085	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.866	0.016	0.000	0.002	-1	1	0.239	0.004	1.000	7.806	7.807	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.005	0.000	0.006	-1	1	3.718	0.014	0.922	0.002	0.002	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.716	0.017	0.000	0.003	-1	5	3.753	0.007	0.896	0.724	0.724	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.009	0.002	0.000	0.009	-1	5	0.238	0.004	1.000	0.037	0.037	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.844469	0.976965	NaN	...	kd_tree	-1	1	0.929	2.582057	0.222149	1.000	0.327053	0.328261	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.002536	0.000441	NaN	...	kd_tree	-1	1	1.000	112.735469	0.000000	0.946	0.000022	0.000022	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	1.026248	0.294924	NaN	...	kd_tree	-1	5	0.946	114.529535	0.000000	0.951	0.008961	0.008961	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	6.051497	0.298986	NaN	...	kd_tree	1	100	0.951	2.582627	0.155704	1.000	2.343156	2.347410	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.004794	0.001031	NaN	...	kd_tree	1	100	1.000	114.640923	0.000000	0.946	0.000042	0.000042	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	3.220000	0.165930	NaN	...	kd_tree	-1	100	0.951	115.427974	0.000000	0.929	0.027896	0.027896	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.731796	0.218939	NaN	...	kd_tree	1	5	0.946	0.004967	0.000113	1.000	348.631641	348.722490	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001484	0.000412	NaN	...	kd_tree	1	5	1.000	0.039421	0.000478	0.911	0.037649	0.037652	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	0.940384	0.253248	NaN	...	kd_tree	1	1	0.929	0.060429	0.000117	0.894	15.561717	15.561746	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.024905	0.011997	NaN	...	kd_tree	-1	1	0.891	0.005021	0.000111	1.000	4.959880	4.961100	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.002029	0.000258	NaN	...	kd_tree	-1	1	1.000	0.038964	0.000453	0.911	0.052077	0.052080	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.022914	0.000642	NaN	...	kd_tree	-1	5	0.911	0.038542	0.000291	0.891	0.594509	0.594526	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.520	0.075	0.023	0.0	-1	1	114.436	0.000	0.929	0.031	0.031	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.918	0.057	0.020	0.0	-1	5	2.596	0.175	1.000	1.509	1.513	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.848	0.097	0.021	0.0	1	100	113.991	0.000	0.951	0.034	0.034	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.423	0.076	0.018	0.0	-1	100	2.572	0.167	1.000	1.720	1.723	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	4.274	0.043	0.019	0.0	1	5	0.041	0.010	0.891	103.144	106.387	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.853	0.057	0.021	0.0	1	1	0.005	0.000	1.000	768.119	768.428	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.021	0.0	-1	1	0.061	0.000	0.894	0.012	0.012	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.000	0.032	0.0	-1	5	0.005	0.000	1.000	0.102	0.102	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.844	0.977	0.000	0.001	-1	1	2.582	0.222	1.000	0.327	0.328	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.000	0.000	0.003	-1	1	112.735	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.026	0.295	0.000	0.001	-1	5	114.530	0.000	0.951	0.009	0.009	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	2.603	0.184	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	6.051	0.299	0.000	0.006	1	100	2.583	0.156	1.000	2.343	2.347	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	1	100	114.641	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.220	0.166	0.000	0.003	-1	100	115.428	0.000	0.929	0.028	0.028	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	2.587	0.164	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.732	0.219	0.000	0.002	1	5	0.005	0.000	1.000	348.632	348.722	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	5	0.039	0.000	0.911	0.038	0.038	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.940	0.253	0.000	0.001	1	1	0.060	0.000	0.894	15.562	15.562	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.005	0.000	1.000	0.195	0.195	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.025	0.012	0.001	0.000	-1	1	0.005	0.000	1.000	4.960	4.961	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	0.039	0.000	0.911	0.052	0.052	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.023	0.001	0.001	0.000	-1	5	0.039	0.000	0.891	0.595	0.595	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.005	0.000	1.000	0.399	0.399	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.111521	0.002679	300	...	100	100	10	0.824	0.422142	0.007131	1.0	0.264178	0.264216	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	95.753	0.0	300	0.001	0.001	0.504	0.022	0.824	189.994	190.173	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.112	0.003	300	0.007	0.0	0.422	0.007	1.0	0.264	0.264	See	See